{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/NeuromatchAcademy/course-content/blob/master/tutorials/W0D3_LinearAlgebra/W0D3_Tutorial1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neuromatch Academy: Week 0, Day 3, Tutorial 1\n",
    "# Linear Algebra: Vectors\n",
    "\n",
    "__Content creators:__ Ella Batty\n",
    "\n",
    "\n",
    "__Content reviewers:__ Name Surname, Name Surname. \n",
    "\n",
    "__Content editors:__ Name Surname, Name Surname.\n",
    "\n",
    "__Production editors:__ Siddharth Suresh  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#Tutorial Objectives\n",
    "During today, we will learn the basics of linear algebra, focusing on the topics that underlie the material on future days in the NMA Computational Neuroscience course. In this tutorial, we focus on vectors: their definition, their properties & operations, and how we can use them to define our coordinate system. \n",
    "\n",
    "By the end of this tutorial, you will:\n",
    "* Be able to provide an example of how linear algebra is used in computational neuroscience\n",
    "* Be able to describe vectors, their properties (dimensionality/length), and their operations (scalar multiplication, vector addition, dot product) geometrically \n",
    "* Be able to determine and explain the number of basis vectors necessary for a given vector space\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:18.438086Z",
     "iopub.status.busy": "2021-05-30T19:06:18.437488Z",
     "iopub.status.idle": "2021-05-30T19:06:18.508346Z",
     "shell.execute_reply": "2021-05-30T19:06:18.508791Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 1: Why do we care about linear algebra?\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"epi-j3kleK0\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Setup\n",
    "\n",
    "Execute the following cells to set up the notebook environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:18.512360Z",
     "iopub.status.busy": "2021-05-30T19:06:18.511822Z",
     "iopub.status.idle": "2021-05-30T19:06:19.216757Z",
     "shell.execute_reply": "2021-05-30T19:06:19.216181Z"
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:19.221418Z",
     "iopub.status.busy": "2021-05-30T19:06:19.220734Z",
     "iopub.status.idle": "2021-05-30T19:06:19.303745Z",
     "shell.execute_reply": "2021-05-30T19:06:19.304164Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Figure settings\n",
    "import ipywidgets as widgets       # interactive display\n",
    "from ipywidgets import fixed\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "plt.style.use(\"https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:19.313149Z",
     "iopub.status.busy": "2021-05-30T19:06:19.311873Z",
     "iopub.status.idle": "2021-05-30T19:06:19.313677Z",
     "shell.execute_reply": "2021-05-30T19:06:19.314047Z"
    }
   },
   "outputs": [],
   "source": [
    "# @title Plotting function\n",
    "def visualize_vectors(v, v_unit):\n",
    "  fig, ax = plt.subplots()\n",
    "\n",
    "  ax.spines['top'].set_color('none')\n",
    "  ax.spines['bottom'].set_position('zero')\n",
    "  ax.spines['left'].set_position('zero')\n",
    "  ax.spines['right'].set_color('none')\n",
    "\n",
    "  ax.set(xlim = [-6, 6], ylim = [-6, 6])\n",
    "\n",
    "  ax.grid(alpha=.4)\n",
    "  v_arr = ax.arrow(0, 0, v[0], v[1], width=0.08, color='#3d85c6', length_includes_head = True);\n",
    "  v_unit_arr = ax.arrow(0, 0, v_unit[0], v_unit[1], width=0.08, color='#ea9999', length_includes_head = True);\n",
    "  ax.set(xlim = [-4, 4], ylim = [-4, 4]);\n",
    "\n",
    "  leg = ax.legend([v_arr, v_unit_arr], [r\"Vector $\\mathbf{v}$\", r\"Normalized vector $\\tilde{\\mathbf{v}}$\"], handlelength = 0, fontsize = 20, loc = 'upper left')\n",
    "  for handle, label in zip(leg.legendHandles, leg.texts):\n",
    "    label.set_color(handle.get_facecolor())\n",
    "    handle.set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Section 1: Intro to vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1.1: What is a vector?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:19.319041Z",
     "iopub.status.busy": "2021-05-30T19:06:19.318514Z",
     "iopub.status.idle": "2021-05-30T19:06:19.346970Z",
     "shell.execute_reply": "2021-05-30T19:06:19.346538Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 2: Vector Definition & Properties\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"eN9ZxQvTTxw\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A vector $\\mathbf{x}$ can be considered from at least two perspectives: as an ordered list of numbers or as an arrow with the base at the origin of a coordinate system. These are two ways of looking at a single thing: in the arrow case, the tip of the arrow is defined by a coordinate (which can be represented by the ordered list). \n",
    "<br/><br/>\n",
    "\n",
    "The **dimensionality of a vector** is determined by the number of components in the ordered list (or the dimensionality of the space in which the arrow exists).  For example, $\\mathbf{x} = \\begin{bmatrix}\n",
    "4 \\\\\n",
    "2 \\\\\n",
    "1 \\\\ \n",
    "\\end{bmatrix}$ is a 3 dimensional vector. We can refer to components by $\\mathbf{x}_i$ where i indicates the position of the component. In this vector, $\\mathbf{x_1} = 4$, $\\mathbf{x_2} = 2$, and $\\mathbf{x_3} = 1$. \n",
    "\n",
    "<br/><br/>\n",
    "One defining property of a vector is its length: $||\\mathbf{x}||$. This is the length of the arrow and can be computed as: \n",
    "$$||\\mathbf{x}|| = \\sqrt{\\sum_{i=1}^N \\mathbf{x}_i^2} $$\n",
    "\n",
    "\n",
    "<br/><br/>\n",
    "We have two special types of vectors with specific names. **Zero vectors** have length 0 (and all components equal 0). **Unit vectors** have length 1. You can normalize a vector and create a unit vector, $\\tilde{\\mathbf{x}}$, by dividing by its length,$||\\mathbf{x}||$ :\n",
    "\n",
    "$$\\tilde{\\mathbf{x}} = \\frac{\\mathbf{x}}{||\\mathbf{x}||} $$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coding Exercise 1.1: Normalizing vectors\n",
    "\n",
    "In this exercise, you will first create the below vector, $\\mathbf{v}$, using a numpy array. You will then implement a function, `normalize_vector`, that outputs a normalized version of the input vector (by dividing each component by the vector length).  You will then visualize both the original vector and the unit vector. \n",
    "\n",
    "First think: how do you think the normalized unit vector will compare to the original vector in terms of direction and length?\n",
    "\n",
    "\n",
    "$$\\mathbf{v} = \\begin{bmatrix} 4 \\\\ 1 \\\\ \\end{bmatrix} $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:19.352792Z",
     "iopub.status.busy": "2021-05-30T19:06:19.351430Z",
     "iopub.status.idle": "2021-05-30T19:06:19.353340Z",
     "shell.execute_reply": "2021-05-30T19:06:19.353683Z"
    }
   },
   "outputs": [],
   "source": [
    "def normalize_vector(input_vector):\n",
    "  \"\"\" Normalizes a vector to length 1\n",
    "\n",
    "  Inputs:\n",
    "    input_vector (ndarray): array of shape (n_dim, ) where n_dim is the\n",
    "        dimensionality of the vector\n",
    "\n",
    "  Outputs:\n",
    "    (ndarray): normalized vector of length 1\n",
    "  \"\"\"\n",
    "\n",
    "  #################################################################################\n",
    "  ## TODO for students: complete normalize_vector function ##\n",
    "  # Fill out function and remove\n",
    "  raise NotImplementedError(\"Student exercise: complete normalize_vector function\")\n",
    "  #################################################################################\n",
    "\n",
    "  # Compute vector length (use np.linalg.norm)\n",
    "  vector_length = ...\n",
    "\n",
    "  # Create normalized vector\n",
    "  normalized_vector = ...\n",
    "\n",
    "  return normalized_vector\n",
    "\n",
    "# Create vector v from above (use np.array)\n",
    "v = ...\n",
    "\n",
    "# Uncomment code below when function is implemented to visualize\n",
    "\n",
    "# Call the function\n",
    "# v_unit = normalize_vector(v)\n",
    "\n",
    "# Visualize the vectors\n",
    "# visualize_vectors(v, v_unit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:19.359018Z",
     "iopub.status.busy": "2021-05-30T19:06:19.357520Z",
     "iopub.status.idle": "2021-05-30T19:06:19.839848Z",
     "shell.execute_reply": "2021-05-30T19:06:19.839077Z"
    }
   },
   "outputs": [],
   "source": [
    "# to_remove solution\n",
    "def normalize_vector(input_vector):\n",
    "  \"\"\" Normalizes a vector to length 1\n",
    "\n",
    "  Inputs:\n",
    "    input_vector (ndarray): array of shape (n_dim, ) where n_dim is the\n",
    "        dimensionality of the vector\n",
    "\n",
    "  Outputs:\n",
    "    (ndarray): normalized vector of length 1\n",
    "  \"\"\"\n",
    "\n",
    "  #################################################################################\n",
    "  ## TODO for students: complete normalize_vector function ##\n",
    "  # Fill out function and remove\n",
    "  # raise NotImplementedError(\"Student exercise: complete normalize_vector function\")\n",
    "  #################################################################################\n",
    "\n",
    "  # Compute vector length (use np.linalg.norm)\n",
    "  vector_length = np.linalg.norm(input_vector)\n",
    "\n",
    "  # Create normalized vector\n",
    "  normalized_vector = input_vector / vector_length\n",
    "\n",
    "  return normalized_vector\n",
    "\n",
    "# Create vector v from above (use np.array)\n",
    "v = np.array([4, 1])\n",
    "\n",
    "# Call the function\n",
    "v_unit = normalize_vector(v)\n",
    "\n",
    "# Visualize the vectors\n",
    "with plt.xkcd():\n",
    "  visualize_vectors(v, v_unit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Note that the unit vector still points in the same direction as the original (it overlays it). Dividing each component by a constant (the length) does not change the direction. The only change is the length - the normalized vector has a length of 1. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1.2: Vector operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:19.845386Z",
     "iopub.status.busy": "2021-05-30T19:06:19.844791Z",
     "iopub.status.idle": "2021-05-30T19:06:19.873662Z",
     "shell.execute_reply": "2021-05-30T19:06:19.873230Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 3: Linear Combinations of Vectors\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"nDCgnQWErEQ\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have several mathematical operations we perform on or with vectors. **Scalar multiplication** of a vector changes its length but not its direction (except for sometimes reversing by 180 degrees as we'll see in the next demo). When we multiply a vector, $\\mathbf{x}$,  by a scalar, $a$, we multiply each individual component by the scalar:\n",
    "\n",
    "$$ a\\mathbf{x} = \\begin{bmatrix}\n",
    "    a\\mathbf{x}_1 \\\\ a\\mathbf{x}_2 \\\\ \\vdots \\\\ a\\mathbf{x}_N\n",
    "\\end{bmatrix}$$\n",
    "\n",
    "You can visualize **vector addition** as the stacking of the two arrows head to tail. In essence, you are moving the second vector so its base is at the tip of the first, without changing its direction. The tip of this stacked second vector is the new added vector's tip. In order words, if you visualize two vectors as sides of a parallelogram, adding them will create a vector with the 4th vertex of that parallelogram as its tip. Numerically, this means that we add the corresponding components of each vector:\n",
    "\n",
    "\n",
    " $$\\mathbf{x} + \\mathbf{y} = \\begin{bmatrix}\n",
    "           \\mathbf{x}_{1} + \\mathbf{y}_1 \\\\ \\mathbf{x}_{2} + \\mathbf{y}_2\\\\ \\vdots \\\\ \\mathbf{x}_{N} + \\mathbf{y}_N \n",
    "\\end{bmatrix}$$\n",
    "\n",
    "\n",
    "We call a group of 2 or more vectors a **set of vectors**. A **linear combination** of a set of vectors is a combination of the set using scalar multiplication and vector addition. Essentially, we are multiplying each vector in the set by a scalar and then adding all the scalar multiplied vectors together: the output of this (another vector) is a linear combination of the set. More formally defined, a vector, $\\mathbf{u}$, is a linear combination of a set of vectors $\\mathbf{v}^1, \\mathbf{v}^2, ..., \\mathbf{v}^N$ with (scalar) weights $c_1, c_2, ...c_N$ if:\n",
    "\n",
    "$$\\mathbf{u} = c_1\\mathbf{v}^1 + c_2\\mathbf{v}^2 + ... + c_n\\mathbf{v}^N $$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Demo: Linear combination of vectors\n",
    "In the following demo, we will demonstrate a linear combination of two vectors, $\\mathbf{x}$ and $\\mathbf{y}$. \n",
    "\n",
    "$$\\mathbf{z} = a\\mathbf{x} + b\\mathbf{y}$$\n",
    "\n",
    "where $\\mathbf{x} = \\begin{bmatrix}3 \\\\ 1 \\\\\\end{bmatrix}$ and $\\mathbf{y} = \\begin{bmatrix}-1 \\\\ 2 \\\\\\end{bmatrix}$.\n",
    "\n",
    "\n",
    "You will play with the scalar multiples $a$ and $b$ to visualize both scalar multiplication and vector addition. Think and talk through the following questions:\n",
    "\n",
    "\n",
    "1.   How does $a\\mathbf{x}$ compare to $\\mathbf{x}$ when $a$ is negative? \n",
    "2.   How does $a\\mathbf{x}$ compare to $\\mathbf{x}$ when $a$ is a fraction?\n",
    "3.  Can you get $\\mathbf{z}$ to point to anywhere in the 2D space with combinations of $a$ and $b$?\n",
    "4.  Would this be the case no matter what $\\mathbf{x}$ and $\\mathbf{y}$ are, as long as they are both 2D vectors?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:19.904755Z",
     "iopub.status.busy": "2021-05-30T19:06:19.904270Z",
     "iopub.status.idle": "2021-05-30T19:06:20.181748Z",
     "shell.execute_reply": "2021-05-30T19:06:20.181269Z"
    }
   },
   "outputs": [],
   "source": [
    " #@markdown Make sure you execute this cell to enable the widget! Move the sliders for “a” and “b”. After releasing the slider, be patient for a couple of seconds to see the desired change.\n",
    "\n",
    "\n",
    "def plot_arrows(a_times_x, b_times_y):\n",
    "  fig, ax = plt.subplots(figsize=(10, 7))\n",
    "\n",
    "  ax.spines['top'].set_color('none')\n",
    "  ax.spines['bottom'].set_position('zero')\n",
    "  ax.spines['left'].set_position('zero')\n",
    "  ax.spines['right'].set_color('none')\n",
    "\n",
    "  ax.set_aspect('equal', adjustable='box')\n",
    "  ax.set(xlim = [-6, 6], ylim = [-6, 6], xticks = np.arange(-6, 6), yticks = np.arange(-6, 6), xticklabels = [], yticklabels = [])\n",
    "\n",
    "  ax.grid(alpha=.4)\n",
    "\n",
    "  z = a_times_x + b_times_y\n",
    "  z_arr = ax.arrow(0, 0, z[0], z[1], width=.08, color='k', length_includes_head = True);\n",
    "\n",
    "  ax_arr = ax.arrow(0, 0, a_times_x[0], a_times_x[1], width=.08, color='#01DFD7', length_includes_head = True);\n",
    "  by_arr = ax.arrow(0, 0, b_times_y[0], b_times_y[1], width=.08, color='#0080FF', length_includes_head = True);\n",
    "\n",
    "  ax.plot([a_times_x[0], z[0]], [a_times_x[1], z[1]], '--k')\n",
    "  ax.plot([b_times_y[0], z[0]], [b_times_y[1], z[1]], '--k')\n",
    "\n",
    "  leg = ax.legend([ax_arr, by_arr, z_arr], [r\"$a\\mathbf{x}$\", r\"$b\\mathbf{y}$\", r\"$\\mathbf{z} = a\\mathbf{x} + b\\mathbf{y}$\"], handlelength = 0, fontsize = 25, loc = 'center left', bbox_to_anchor=(1.05, .5))\n",
    "  for handle, label in zip(leg.legendHandles, leg.texts):\n",
    "        label.set_color(handle.get_facecolor())\n",
    "        handle.set_visible(False)\n",
    "\n",
    "@widgets.interact(a = widgets.FloatSlider(value=1.0, min=-2, max=2, step=0.1), b = widgets.FloatSlider(value=1.0, min=-2, max=2, step=0.1))\n",
    "def plot_linear_combination(a, b):\n",
    "  x = np.array([3, 1])\n",
    "  y = np.array([-1, 2])\n",
    "\n",
    "  plot_arrows(a*x, b*y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.188943Z",
     "iopub.status.busy": "2021-05-30T19:06:20.188460Z",
     "iopub.status.idle": "2021-05-30T19:06:20.192477Z",
     "shell.execute_reply": "2021-05-30T19:06:20.191550Z"
    }
   },
   "outputs": [],
   "source": [
    "# to_remove explanation\n",
    "\n",
    "# 1. When a is negative, the direction of ax is the exact opposite of x\n",
    "# 2. When a is a positive fraction, ax is the same direction but shorter than x\n",
    "# 3. Yes you can get z to point anywhere in 2D space by changing a and b around\n",
    "# 4. No, if x and y pointed in the same direction, you could not get anywhere in\n",
    "#    2D space. We will delve into this in the next section!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Section 2: Defining space through vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2.1: Span & Linear Independence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.197674Z",
     "iopub.status.busy": "2021-05-30T19:06:20.197114Z",
     "iopub.status.idle": "2021-05-30T19:06:20.225493Z",
     "shell.execute_reply": "2021-05-30T19:06:20.225068Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 4: Span and Linear Independence\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"izFyjUfVW10\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **span of a set of vectors** is the set of all possible linear combinations of those vectors. In the last demo, you saw that the vector $\\mathbf{x}$ and the vector $\\mathbf{y}$ spanned 2D space (${\\rm I\\!R}^2$). This means you can get to any point in 2D space (corresponding to vector $\\mathbf{v}$ with some combination of $a$ and $b$, the scalar multiples in a linear combination of $\\mathbf{x}$ and $\\mathbf{y}$:\n",
    "\n",
    "$$\\mathbf{v} = a\\mathbf{x} + b\\mathbf{y}$$\n",
    "\n",
    "Even more specifically, in this situation, each point in 2D space is identified by a unique and single combination of $a$ and $b$: if you know the point, you know what $a$ and $b$ are. \n",
    "\n",
    "A set of vectors in linearly dependent if one can be written as a linear combination of the others. If this is not the case, the set of vectors is linearly dependent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Think! 2.1: Determing dependence\n",
    "\n",
    "Let's say we have four vectors: \n",
    "\n",
    "$$ \\mathbf{a} = \\begin{bmatrix}\n",
    "    4 \\\\ 2 \\\\ 0\n",
    "\\end{bmatrix},  \\mathbf{b} = \\begin{bmatrix}\n",
    "    1 \\\\ 0 \\\\ 3, \n",
    "\\end{bmatrix}, \\mathbf{c} = \\begin{bmatrix}\n",
    "    5 \\\\ 2 \\\\ 3\n",
    "\\end{bmatrix}, \\mathbf{d} = \\begin{bmatrix}\n",
    "    3 \\\\ 3 \\\\ 2\n",
    "\\end{bmatrix}$$\n",
    "\n",
    "1. Are these vectors linearly independent or dependent? Why?\n",
    "2. What is the span of the set of vectors $\\{\\mathbf{a}, \\mathbf{b}, \\mathbf{c}, \\mathbf{d}\\}$? \n",
    "3. What is the span of a set just consisting of vector $\\{\\mathbf{a}$} (in general terms)?\n",
    "4. What is the span of the set of vectors $\\{\\mathbf{a}, \\mathbf{b}$} (in general terms)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.228819Z",
     "iopub.status.busy": "2021-05-30T19:06:20.228278Z",
     "iopub.status.idle": "2021-05-30T19:06:20.232088Z",
     "shell.execute_reply": "2021-05-30T19:06:20.231664Z"
    }
   },
   "outputs": [],
   "source": [
    "# to_remove explanation\n",
    "\n",
    "# 1) They are linearly dependent as one can be formed as a linear combination of\n",
    "#     the others (a + b = c). You could also have known this because you have four\n",
    "#.    3D vectors. You don't need 4 vectors to get anywhere in 3D space!\n",
    "#.\n",
    "# 2) The span of a, b, c, and d is all of 3D space (R^3).\n",
    "\n",
    "# 3) The span of a and b is a 1D line through 3D space (note this is not R^1)\n",
    "\n",
    "# 4) The span of a and b is a 2d plane through 3D space (note this is not R^2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2.2: Basis vectors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.236703Z",
     "iopub.status.busy": "2021-05-30T19:06:20.236181Z",
     "iopub.status.idle": "2021-05-30T19:06:20.266097Z",
     "shell.execute_reply": "2021-05-30T19:06:20.267437Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 5: Basis vectors\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"f03YDxUUlXs\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a set of vectors spans 2D space are linearly independent, they form a **basis** for 2D space.  This means that we can identify all points in 2D space with reference to these two vectors, instead of to the traditional x-axis and y-axis coordinates. In fact, you've been using a basis all along when you identify points by their x-axis and y-axis coordinates: the **standard basis**! In 2D space, the standard basis vectors ($\\tilde{\\mathbf{e}}_1$ and $\\tilde{\\mathbf{e}}_2$) are the unit vectors corresponding to the x and y axes:\n",
    " \n",
    "$$\\tilde{\\mathbf{e}}_1 = \\begin{bmatrix} 1\\\\ 0\\\\ \\end{bmatrix} ,\\tilde{\\mathbf{e}}_2 = \\begin{bmatrix} 0\\\\ 1\\\\ \\end{bmatrix} $$\n",
    "\n",
    "When you refer to a point on a 2D plot as (4, 2), you are really identifying the scalar multiples to get to that point as a linear combination of the standard basis. The standard basis is convenient - and what we're used to - but it is essential for a variety of applications of linear algebra (including some we'll encounter in this course) to be able to switch our frame of reference to a different basis. In this example, we can now refer to points as (a, b) where a and b are the scalar multiples to get to that point as a linear combination of $\\mathbf{x}$ and $\\mathbf{y}$.\n",
    "\n",
    "Let's return to this idea of linear independence of vectors.  \n",
    "\n",
    "I've been referring to 2D space a lot but there are lots of **vector spaces** for which we can use basis vectors. We'll leave the formal definition of vector spaces to the bonus material as it's a little unwieldy but basically it's a set of vectors where all linear combinations of all vectors are included in the set. 1D space (${\\rm I\\!R}^1$), 2D space (${\\rm I\\!R}^2$), 3D space (${\\rm I\\!R}^3$), and so on are all vector spaces. These are not the only ones though: subsets of these vector spaces can also be vector spaces themselves. For example, a 1D line through a 2D plane is a vector space.  And we have even more unusual vector spaces - I give an example of one in the bonus material.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Think! 2.2: Figuring out a basis\n",
    "\n",
    "Let's say we are looking at the firing rates of 3 neurons. It turns out that due to their tightly-linked interactions, they always fire in such a way that the point in 3D space falls along a 2D plane (that passes through the origin) as represented by the blue plane in the figure below. This 2D plane is a subspace of all of 3-D space (R3).\n",
    "\n",
    "1) How many basis vectors would you need for the 2D plane? How many components would each basic vector have?\n",
    "\n",
    "2) How many numbers would you need to be able to fully describe the state of the 3 neuron population at a given time?\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.319050Z",
     "iopub.status.busy": "2021-05-30T19:06:20.316545Z",
     "iopub.status.idle": "2021-05-30T19:06:20.520117Z",
     "shell.execute_reply": "2021-05-30T19:06:20.519643Z"
    }
   },
   "outputs": [],
   "source": [
    "# @markdown Execute this cell to see figure\n",
    "\n",
    "fig = plt.figure()\n",
    "\n",
    "# add axes\n",
    "ax = fig.add_subplot(111,projection='3d')\n",
    "\n",
    "xx, yy = np.meshgrid(range(-4, 4), range(-4, 4))\n",
    "z = .7*xx + -.2*yy\n",
    "\n",
    "# plot the plane\n",
    "ax.plot_surface(xx, yy, z, alpha=0.5)\n",
    "\n",
    "ax.set(xlim = [0, 5], ylim = [5, 0], zlim = [0, 5], xlabel = 'Neuron 1 Firing Rate',  ylabel = 'Neuron 2 Firing Rate', zlabel = 'Neuron 3 Firing Rate');\n",
    "#ax.invert_xaxis()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.524712Z",
     "iopub.status.busy": "2021-05-30T19:06:20.523581Z",
     "iopub.status.idle": "2021-05-30T19:06:20.525277Z",
     "shell.execute_reply": "2021-05-30T19:06:20.525649Z"
    }
   },
   "outputs": [],
   "source": [
    "# to_remove explanation\n",
    "\n",
    "# 1. You would need 2 basis vectors as the true dimensionality of the 2D plane is 2.\n",
    "# Each vector would have 3 components. The two basic vectors would be independent\n",
    "# vectors that lie in the 2D plane.\n",
    "\n",
    "# 2. You can describe the state of the 3 neuron population activity with 2 numbers by\n",
    "# using the new basis vectors as the coordinate system. We can now describe the neural\n",
    "# activity in a lower dimensional space because there is lower-dimensional structure.\n",
    "# We'll learn much more about this, and lots of related interesting research, on the\n",
    "# dimensionality reduction day in the comp neuro course!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 3: The dot product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3.1: Neural modeling using the dot product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.530881Z",
     "iopub.status.busy": "2021-05-30T19:06:20.530345Z",
     "iopub.status.idle": "2021-05-30T19:06:20.560253Z",
     "shell.execute_reply": "2021-05-30T19:06:20.561103Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 6: The dot product\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"i5pQKRW8Etc\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dot product between two vectors is a scalar value computed as the sum of element-wise multiplication of vector components. \n",
    "$$\\text{Dot product of } \\mathbf{x} \\text{ and } \\mathbf{y} = \\mathbf{x} \\cdot \\mathbf{y} = \\sum_{i=1}^{N} \\mathbf{x}_i\\mathbf{y}_i $$\n",
    "\n",
    "If we have vector $\\mathbf{x} = \\begin{bmatrix}2\\\\ 3\\\\\\end{bmatrix}$ and $\\mathbf{x} = \\begin{bmatrix}1\\\\ 4\\\\\\end{bmatrix}$, then:\n",
    "$$\\mathbf{x} \\cdot \\mathbf{y} = 2*1 + 3*4 = 14$$.\n",
    "\n",
    "In code, we can compute the dot product between two vectors represented by numpy arrays using `np.dot`:\n",
    "\n",
    "\n",
    "```\n",
    "x = np.array([2, 3])\n",
    "y = np.array([1, 4])\n",
    "dot_prod = np.dot(x, y)\n",
    "```\n",
    "\n",
    "\n",
    " We will discuss more about what the dot product represents geometrically after the next demo.\n",
    "\n",
    "\n",
    "We are using the dot product to model neural firing. Specifically, we have multiple presynaptic neurons connected to a postsynaptic neuron. The postsynaptic neuron response is a weighted sum of the presynaptic firing rates (where the weights are the strengths of the synapses). We can capture this as the dot product between a vector of presynaptic firing rates and a vector of weights from each presynaptic neuron. \n",
    "\n",
    "In the following demo, we have two presynaptic neurons with varying firing rates ($a_1$ and $a_2$). We will allow our firing rates to be negative in this example. We can represent our presynaptic firing rates with the vector $\\mathbf{a} = \\begin{bmatrix} a_1\\\\ a_2\\\\ \\end{bmatrix}$.\n",
    "\n",
    "We have weights from each of these to a postsynaptic neuron: the weight of the connection from the first presynaptic neuron is 1 and the weight from the second is 2. See the circuit below. We can represent our weights with the vector $\\mathbf{w} = \\begin{bmatrix} 1\\\\ 2\\\\ \\end{bmatrix}$.\n",
    "\n",
    "The postsynaptic firing rate is the dot product of the presynaptic firing rate vector and the weight vector:\n",
    "\n",
    "$$b = \\mathbf{w}\\cdot\\mathbf{a} = w_1a_1 + w_2a_2$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Screen Shot 2021-05-28 at 3.55.08 AM.png]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Demo: Postsynaptic firing\n",
    "\n",
    "\n",
    "In the following demo you will try various combinations of firing rates of the pre-synaptic neurons (thus moving the vector that corresponds to the firing rates).  When you move the presynaptic firing rates around, you will see the heatmap in that area appear which shows the corresponding postsynaptic neuron firing rate (for that combo of presynaptic firing). Remember that this value is computed as the dot product between the presynaptic firing rate vector and the weight vector.\n",
    "\n",
    "Play with the following demo and then discuss these questions:\n",
    "\n",
    "\n",
    "1.   For a given length of the presynaptic firing rate vector, what direction maximally excites the postsynaptic neuron (highest postsynaptic firing rate)? How does it relate to the weights vector?\n",
    "2.   For a given length of the presynaptic firing rate vector, what direction minimally excites the postsynaptic neuron? \n",
    "3.   When does the postsynaptic neuron have 0 firing rate? What is the relationship between the weights and presynaptic firing rate vector?\n",
    "4.   For a given direction, how does the postsynaptic firing rate vary with length of the presynaptic firing rate vector?\n",
    "5.   Let's say the presynaptic neurons are tightly linked and the length of the presynaptic firing rate has to be 1. In other words, it has to be a unit vector. How would you find the presynaptic firing rates that maximally excite the postsynaptic neuron?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.574947Z",
     "iopub.status.busy": "2021-05-30T19:06:20.563916Z",
     "iopub.status.idle": "2021-05-30T19:06:20.938206Z",
     "shell.execute_reply": "2021-05-30T19:06:20.937643Z"
    }
   },
   "outputs": [],
   "source": [
    "#@markdown Make sure you execute this cell to enable the widget!\n",
    "\n",
    "step = .1\n",
    "x_vec = np.arange(-4, 4.001, step)\n",
    "y_vec = np.arange(-4, 4.001, step)\n",
    "n_pixels = x_vec.shape[0]\n",
    "\n",
    "\n",
    "heatmap = np.zeros((n_pixels, n_pixels))\n",
    "y = np.array([1, 3])\n",
    "for i, coord1 in enumerate(x_vec):\n",
    "  for j, coord2 in enumerate(x_vec):\n",
    "    heatmap[i, j] = np.dot(np.array([coord1, coord2]), y)\n",
    "circle_mask = np.zeros((n_pixels, n_pixels))\n",
    "\n",
    "for i, coord_i in enumerate(x_vec):\n",
    "  for j, coord_j in enumerate(y_vec):\n",
    "    circle_mask[i, j] = np.sqrt(coord_i**2 + coord_j**2)\n",
    "\n",
    "circle_mask = circle_mask < 4\n",
    "\n",
    "heatmap = heatmap * circle_mask\n",
    "mask = np.zeros((n_pixels, n_pixels))\n",
    "\n",
    "def plot_heatmap(x, y, mask):\n",
    "\n",
    "  fig, ax = plt.subplots()\n",
    "\n",
    "  ax.spines['top'].set_color('none')\n",
    "  ax.spines['bottom'].set_position('zero')\n",
    "  ax.spines['left'].set_position('zero')\n",
    "  ax.spines['right'].set_color('none')\n",
    "\n",
    "  heatmap[np.where(x_vec == x[0])[0], np.where(x_vec == x[1])[0]] = np.dot(x, y)\n",
    "\n",
    "  masked_x = np.abs(x_vec - x[0]) < 1\n",
    "  masked_y = np.abs(y_vec - x[1]) < 1\n",
    "  mask += np.outer(masked_x, masked_y)\n",
    "  mask = np.minimum(mask, 1)\n",
    "\n",
    "  im = ax.imshow((heatmap * mask).T, vmin = -15, vmax = 15, origin = 'lower', alpha = .5, extent=[-n_pixels/2., n_pixels/2., -n_pixels/2., n_pixels/2. ], cmap = 'bwr')\n",
    "  cbar = plt.colorbar(im, ax = ax)\n",
    "  cbar.set_label('Response of postsynaptic neuron', rotation=270, labelpad=20)\n",
    "  ax.set(xticklabels = [], yticklabels = [], xlabel = 'Neuron 1')\n",
    "  ax.set_ylabel('Neuron 2', rotation=0)\n",
    "\n",
    "  ax.xaxis.set_label_coords(0, .45)\n",
    "  ax.yaxis.set_label_coords(.5, 1)\n",
    "  fr_arr = ax.arrow(0, 0, (1/step)*x[0], (1/step)*x[1], width=.5, color='g', length_includes_head = True);\n",
    "  we_arr = ax.arrow(0, 0, (1/step)*y[0], (1/step)*y[1], width=.5, color='k', length_includes_head = True);\n",
    "\n",
    "\n",
    "  leg = ax.legend([fr_arr, we_arr], ['Presynaptic firing rate vector', 'Weight vector'],\n",
    "                  handlelength = 0, frameon=False, fontsize = 17,loc = 'center',\n",
    "                  bbox_to_anchor=(.5, -.1))\n",
    "  for handle, label in zip(leg.legendHandles, leg.texts):\n",
    "        label.set_color(handle.get_facecolor())\n",
    "        handle.set_visible(False)\n",
    "\n",
    "  return mask\n",
    "\n",
    "style = {'description_width': 'initial'}\n",
    "\n",
    "@widgets.interact(neuron1_firing = widgets.FloatSlider(value=1.0, min=-4, max=4, step=1, style=style), neuron2_firing = widgets.FloatSlider(value=1.0, min=-4, max=4, step=1, style=style), mask = fixed(mask),  heatmap = fixed(heatmap))\n",
    "def plot_linear_combination(neuron1_firing, neuron2_firing, mask):\n",
    "  firing_rates = np.array([neuron1_firing, neuron2_firing])\n",
    "  weights = np.array([1, 2])\n",
    "  mask = plot_heatmap(firing_rates, weights, mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.943486Z",
     "iopub.status.busy": "2021-05-30T19:06:20.942406Z",
     "iopub.status.idle": "2021-05-30T19:06:20.944094Z",
     "shell.execute_reply": "2021-05-30T19:06:20.944519Z"
    }
   },
   "outputs": [],
   "source": [
    "# to_remove explanation\n",
    "\n",
    "# 1) The direction of the presynaptic firing rate vector that maximially excites\n",
    "#.   the postsynaptic neuron is the direction of the weight vector.\n",
    "\n",
    "# 2) The direction of the presynaptic firing rate vector that mimimally excites\n",
    "#.   the postsynaptic neuron is the opposite direction of the weight vector.\n",
    "\n",
    "# 3) The postsynaptic neuron has 0 firing rate when the firing rate vector is\n",
    "#.   perpendicular to the weight vector\n",
    "\n",
    "\n",
    "#. 4) The absolute value of the postsynaptic firing rate grows bigger as the length\n",
    "#.   of the presynaptic firing rate increase.\n",
    "\n",
    "\n",
    "#. 5) You want a unit vector in the direction of the weight vector. You can just normalize\n",
    "#.   the weight vector (divide by it's length), as we saw in Section 1!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3.2: The geometry of the dot product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:20.949794Z",
     "iopub.status.busy": "2021-05-30T19:06:20.949269Z",
     "iopub.status.idle": "2021-05-30T19:06:20.979408Z",
     "shell.execute_reply": "2021-05-30T19:06:20.978908Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 7: The geometry of the dot product\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"LGclCl15Y6A\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Summary\n",
    "\n",
    "Hopefully, this tutorial has helped you better understand the world of vectors. Specifically, you learned: \n",
    "\n",
    "* A geometrical view of vectors, their properties, and their operations\n",
    "\n",
    "* How we can define space in terms of basis vectors\n",
    "\n",
    "We will build on this knowledge in the next tutorial when we tackle matrices. \n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "W0D3_Tutorial1",
   "provenance": [],
   "toc_visible": true
  },
  "kernel": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
